Information processing apparatus, information processing method, and program for displaying a virtual object on a display

ABSTRACT

An apparatus including an image processor configured to receive a video including an object, determine a positional relationship between the apparatus and the object, and change a positional relationship between an image superimposed on the video and the object when the positional relationship between the apparatus and the object changes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application U.S. application Ser. No.14/344,135, filed on Mar. 11, 2014, currently pending, which is aNational Stage of PCT/JP12/05904, filed Sep. 14, 2012, which claims thebenefit of priority under 35 U.S.C. § 119 from Japanese PatentApplication No. 2011-247568, filed Nov. 11, 2011. The entire contents ofall of the above are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a program encoded on anon-transitory computer readable medium.

BACKGROUND ART

In recent years, a technique called augmented reality (AR) in whichadditional information is superimposed on a real image so as to bepresented to a user has attracted attention. In the AR technique,information presented to a user is also called annotation and theinformation is visualized as various types of virtual objects such as,for example, texts, icons or animation. As an example of the ARtechnique, Patent Literature 1 discloses a technique in which an imageof a virtual object imitating a real object such as furniture issuperimposed on a real image so as to be displayed, thereby easilyattempting to arrange furniture and so on.

CITATION LIST Patent Literature

[PTL 1]

JP 2003-256876A

SUMMARY Technical Problem

The above-described AR technique is realized, for example, by capturinga real image using a device held by the user and by superimposing thedisplay of a virtual object related to a real object reflected in theimage. In this case, for example, postures of the reflected real objectare variously changed depending on the movement of the user or changesin methods of holding the device. That is to say, even in the same realobject, a size or angle in the image is changed depending on thecircumstances. Thus, even if a captured image of the same real object isdisplayed, there are cases where appropriate display methods ofadditional information are different.

Therefore, it is desirable to provide novel and improved informationprocessing apparatus, information processing method, and program encodedon a non-transitory computer readable medium, capable of appropriatelydisplaying additional information so as to be suitable for a posture ofa real object.

Solution to Problem

In one embodiment, the present invention includes apparatus including animage processor configured to receive a video including an object,determine a positional relationship between the apparatus and theobject, and change a positional relationship between an imagesuperimposed on the video and the object when the positionalrelationship between the apparatus and the object changes.

Advantageous Effects of Invention

As described above, according to embodiments of the present disclosure,it is possible to appropriately display additional information so as tobe suitable for a posture of a real object.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a reference technique regarding a firstembodiment of the present disclosure;

FIG. 2 is a block diagram illustrating an example of the functionalconfiguration of the information processing apparatus according to thefirst embodiment of the present disclosure;

FIG. 3 is a diagram illustrating a first state where a virtual object isdisplayed by the information processing apparatus according to the firstembodiment of the present disclosure;

FIG. 4 is a diagram illustrating a second state where a virtual objectis displayed by the information processing apparatus according to thefirst embodiment of the present disclosure;

FIG. 5 is a diagram illustrating a change in display of a virtual objectby the information processing apparatus according to the firstembodiment of the present disclosure;

FIG. 6 is a flowchart illustrating an example of the process flow ofimage display process in the information processing apparatus accordingto the first embodiment of the present disclosure;

FIG. 7 is a flowchart illustrating an example of the process flow ofdisplay target setting process in the information processing apparatusaccording to the first embodiment of the present disclosure;

FIG. 8 is a diagram illustrating an example of the alternative selectionand the message display according to the first embodiment of the presentdisclosure;

FIG. 9 is a diagram illustrating a first state where a virtual object isdisplayed by an information processing apparatus according to a secondembodiment of the present disclosure;

FIG. 10 is a diagram illustrating a second state where a virtual objectis displayed by the information processing apparatus according to thesecond embodiment of the present disclosure;

FIG. 11 is a diagram illustrating a change in display of a virtualobject by the information processing apparatus according to the secondembodiment of the present disclosure;

FIG. 12 is a diagram illustrating a positional relationship between adetected real object and a device according to a third embodiment of thepresent disclosure;

FIG. 13 is a diagram illustrating a first state where a virtual objectis displayed by the information processing apparatus according to thethird embodiment of the present disclosure;

FIG. 14 is a diagram illustrating a second state where a virtual objectis displayed by the information processing apparatus according to thethird embodiment of the present disclosure;

FIG. 15 is a diagram illustrating a change in display of a virtualobject by the information processing apparatus according to the firstembodiment of the present disclosure;

FIG. 16 is a flowchart illustrating an example of the process flow ofdisplay target setting process in the information processing apparatusaccording to the third embodiment of the present disclosure;

FIG. 17 is a block diagram illustrating another example of thefunctional configuration of an information processing apparatusaccording to an embodiment of the present disclosure;

FIG. 18 is a block diagram illustrating further another example of thefunctional configuration of the information processing apparatusaccording to an embodiment of the present disclosure; and

FIG. 19 is a block diagram illustrating a hardware configuration of theinformation processing apparatus.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

In addition, the description will be made in the following order.

1. First Embodiment (an example of changing a display position relativeto a real object)

2. Second Embodiment (an example of changing an amount of displayedinformation)

3. Third Embodiment (an example of changing a content of displayedinformation)

4. Other Embodiments regarding Apparatus Configuration

5. Supplement

1. First Embodiment

The first embodiment of the present disclosure will be described withreference to FIGS. 1 to 8. In the first embodiment of the presentdisclosure, a display position of a virtual object relative to a realobject is changed depending on a relative positional relationshipbetween the real object reflected in an image and a device capturing theimage.

1-1. Reference Technique

First, a description will be made of a reference technique for betterunderstanding of the present embodiment. FIG. 1 is a diagramillustrating the reference technique regarding the first embodiment ofthe present disclosure.

An information processing apparatus 10 related to the referencetechnique is shown in FIG. 1. The information processing apparatus 10includes an image pickup unit (not shown) and a display unit 12, and theinformation processing apparatus 10 executes an AR application.

Here, the display unit 12 displays an image where the informationprocessing apparatus 10 superimposes a virtual object obj_v related to areal object obj_r on an image in which the real object obj_r captured bythe image pickup unit is reflected. In the shown example, in a casewhere the real object obj_r (a book) is captured from the front side asshown in (a) of FIG. 1, and in a case where the real object obj_r iscaptured obliquely as shown in (b) of FIG. 1, the virtual object obj_v(an image regarding a content of the book) is transformed into differentforms and is displayed. In other words, the virtual object obj_v istransformed and displayed to be suitable for the change in theappearance due to the change in a posture of the real object obj_r.

As such, in the AR application, there are cases where a virtual objectis transformed to be suitable for a posture of a real object and isdisposed in an AR space. Thereby, for example, a user recognizes thevirtual object as if the virtual object is reflected on a surface of areal object and exists therein, and thus can easily understand arelationship between the real object and the virtual object.

However, as shown in (b) of FIG. 1 of the illustrated example, there isa case where it is difficult for the transformed virtual object obj_v tobe visually recognized according to a posture of the real object obj_r.In this case, it is difficult to say that the virtual object obj_v isdisplayed in an appropriate manner.

Therefore, in the present embodiment, by changing a display position ofa virtual object relative to a real object depending on a positionalrelationship between the real object reflected in an image and a devicecapturing the image, even in the case, it is possible to display thevirtual object obj_v in an appropriate manner.

1-2. Apparatus Configuration

Next, with reference to FIG. 2, a functional configuration according tothe present disclosure will be described. FIG. 2 is a block diagramillustrating an example of the functional configuration of theinformation processing apparatus according to the first embodiment ofthe present disclosure.

Referring to FIG. 2, an information processing apparatus 100 accordingto the present embodiment includes an image pickup unit 102, an imagerecognition unit 104, a model DB 106, a positional relationshipdetection unit 108, a selection unit 110, an object DB 112, a displaytarget information output unit 114, a message information output unit116, a display control unit 118, and a display unit 120.

In the present embodiment, the information processing apparatus 100 willbe described as a terminal apparatus into which the respective unitsincluding the image pickup unit and the display unit, such as a smartphone, a tablet type PC (Personal Computer), PDA (Personal DigitalAssistant), a portable gaming machine, or a portable music player areintegrated. However, the information processing apparatus 100 may beother information processing apparatuses such as a notebook type ordesktop type PC. In this case, for example, the image pickup unit or thedisplay unit may be provided separately from the other units. Inaddition, in a case where the information processing apparatus 100 isdivided into a plurality of units, the “information processing apparatus100” in the disclosure such as a “positional relationship between theinformation processing apparatus 100 and a real object” or an “angle ofa real object relative to the information processing apparatus 100”indicates a portion at which the image pickup unit of the informationprocessing apparatus 100 is provided unless otherwise specificallydefined herein.

Among the above-described functional configurations, the image pickupunit 102 is realized by, for example, an image pickup device included inthe information processing apparatus 100. In addition, the imagerecognition unit 104, the positional relationship detection unit 108,the selection unit 110, the display target information output unit 114,the message information output unit 116, and the display control unit118 are realized by, for example, a CPU (Central Processing Unit), a RAM(Random Access Memory), and a ROM (Read Only Memory) of the informationprocessing apparatus 100 being operated according to a program. Themodel DB 106 and the object DB 112 are realized by, for example, aninternal or external storage device of the information processingapparatus 100. The display unit 120 is realized by, for example, avariety of displays which are included in the information processingapparatus 100 as output devices or are connected to the informationprocessing apparatus 100 as external connection devices.

<Image Pickup Unit>

The image pickup unit 102 captures a real space to generate an image.The image pickup unit 102 may capture moving images. The moving imagesare configured by a plurality of images forming consecutive frames. Theimage pickup unit 102 provides the generated image to both the imagerecognition unit 104 and the display control unit 118. At this time, thegenerated image may be temporarily stored in, for example, the RAM orthe storage device.

<Image Recognition Unit>

The image recognition unit 104 performs object recognition using datastored in the model DB 106. The model DB 106 is a database which storesin advance model data regarding shapes or appearances of objects whichare targets recognized by the information processing apparatus 100. Inthe present embodiment, targets recognized by the information processingapparatus 100 are any objects in a real space. The model data includesdata defining a shape of each object, image data such as a predeterminedsymbol mark or a text label added to each object, data of a featureamount set extracted from an existing image for each object, and thelike.

More specifically, the image recognition unit 104 uses an image capturedby the image pickup unit 102 as an input image, thereby recognizingwhich object is reflected in the input image. The image recognition unit104 compares, for example, a feature point set extracted from the inputimage with a shape of an object defined by the model data. In addition,the image recognition unit 104 may compare image data such as a symbolmark or a text label defined by the model data with the input image.Furthermore, the image recognition unit 104 may compare a feature amountof an image of an existing object defined by the model data with afeature amount extracted from the input image.

<Positional Relationship Detection Unit>

The positional relationship detection unit 108 detects a positionalrelationship between the object recognized by the image recognition unit104 and the information processing apparatus 100 (more specifically, theimage pickup unit 102). In the present embodiment, an angle of theobject relative to the information processing apparatus 100 is used as apositional relationship between the object and the informationprocessing apparatus 100. This angle is expressed as, for example, atilt from a reference posture which is set for an object in advance. Inother words, in the present embodiment, the positional relationshipdetection unit 108 detects a posture relationship between the object andthe information processing apparatus 100 as the positional relationshipbetween the object and the information processing apparatus 100.

More specifically, the positional relationship detection unit 108 usesan image captured by the image pickup unit 102 as an input image,thereby detecting a posture of an object reflected in the input image.Here, in the present embodiment, a posture of the object is collectivelyexpressed by a 4*4 homogeneous transformation matrix indicatingtransformation between a model coordinate system in the model datastored in the model DB 106 and a coordinate system of the objectreflected in the input image. The positional relationship detection unit108 extracts an angle of the object relative to the informationprocessing apparatus 100 from the homogeneous transformation matrix.

In addition, detailed examples of the positional relationship acquiredby the positional relationship detection unit 108 will be describedlater. Since a process performed by the positional relationshipdetection unit 108 is common to a process performed by the imagerecognition unit 104 such as referring to the model DB 106, the processmay be performed along with the process performed by the imagerecognition unit 104.

<Selection Unit>

The selection unit 110 selects a display target object which issuperimposed on an image, from a virtual object group including aplurality of kinds of virtual objects related to the real object,according to the positional relationship detected by the positionalrelationship detection unit 108. More specifically, the selection unit110 selects a virtual object from a virtual object group including aplurality of virtual objects of which the display positions relative tothe real object are different from each other, according to the angle ofthe real object relative to the information processing apparatus 100.The virtual object is displayed at a position which is easily visuallyrecognized by a user. The angle of the real object relative to theinformation processing apparatus 100 is acquired by the positionalrelationship detection unit 108. Such a virtual object and an example ofthe selection of a virtual object by the selection unit 110 will bedescribed later.

In the present embodiment, data of the virtual object group is stored inthe object DB 112. The object DB 112 may be a database which stores inadvance data of a virtual object group corresponding to each realobject. In addition, the object DB 112 may be a database whichtemporarily stores a virtual object group which corresponds to a realobject recognized by the image recognition unit 104 and is downloadedfrom an external device.

Furthermore, the object DB 112 may be a database which temporarilystores a virtual object which is obtained by processing a part of or theoverall real object reflected in an image captured by the image pickupunit 102. The processing of the real object is performed by a processingunit (not shown). The processing unit may generate a virtual object byprocessing an image according to a result of a selection made by theselection unit 110. In this case, for example, the selection unit 110refers to information indicating “what kind of virtual object can begenerated” stored in the object DB 112. The selection unit 110 alsoselects the kind of virtual object to be displayed and instructs theprocessing unit to generate the virtual object.

In addition, as described later as an example of the alternativeselection and the message display, in a case where a selected virtualobject cannot be displayed in an image region, the selection unit 110may make an alternative selection of a virtual object which can bedisplayed in the image region regardless of the selection result.

In addition, the selection unit 110 may adjust a selection result suchthat the same kind of virtual object is selected for a plurality of realobjects related to each other. In this case, for example, the selectionunit 110 may average results detected by the positional relationshipdetection unit 108 for the respective real objects. The selection unit110 also may use the averaged result as a positional relationship commonto the respective real objects. In addition, the selection unit 110 maycount the results detected by the positional relationship detection unit108 for the respective real objects and may use a detection resultdetected from more real objects as a positional relationship common tothe respective real objects. Thereby, for example, the kinds of virtualobjects displayed for a plurality of real objects related to each othercan be unified, and the unification of the virtual objects for the realobjects can be shown to a user.

<Display Target Information Output Unit>

The display target information output unit 114 outputs information ofthe display target object selected by the selection unit 110 to thedisplay control unit 118 such that the information is superimposed onthe image captured by the image pickup unit 102 so as to be displayed.Here, the output information of the display target object may be imagedata itself of a virtual object or may be identification informationsuch as an ID for separately referring to the image data of the virtualobject. Here, in a case of outputting the image data of the virtualobject, the display target information output unit 114 may generate asuperimposed image of the virtual object, for example, according to apositional relationship between the real object and the informationprocessing apparatus 100. The positional relationship is detected by thepositional relationship detection unit 108. The display targetinformation output unit 114 also may output information of thesuperimposed image to the display control unit 118. For this process,the display target information output unit 114 may refer to the objectDB 112.

<Message Information Output Unit>

The message information output unit 116 is additionally provided. Themessage information output unit 116 generates information of a messagewhich is superimposed on an image and displayed, and the messageinformation output unit 116 outputs the information to the displaycontrol unit 118. For example, in a case where an alternative displaytarget object is selected on account of restriction of a region of animage captured by the image pickup unit 102, as described later, themessage information output unit 116 generates information of a messageprompting a user to change a region of the image by moving the imagepickup unit 102, and the message information output unit 116 outputs theinformation to the display control unit 118.

<Display Control Unit>

The display control unit 118 superimposes the virtual object which is adisplay target object on the image captured by the image pickup unit102. The superimposition uses the information outputted from the displaytarget information output unit 114. The display control unit 118 alsocauses the display unit 120 to display the superimposed result as an ARapplication image. At this time, the display control unit 118 maygenerate a superimposed image where the display target object isdisposed and transformed to be suitable for the position, posture andsize of the real object reflected in the image, and the display controlunit 118 may superimpose the superimposed image on the image. Asdescribed above, the superimposed image may be generated by the displaytarget information output unit 114. When the superimposed image isgenerated, the display control unit 118 may use, for example,information indicating the positional relationship between the realobject and the information processing apparatus 100. The positionalrelationship is detected by the positional relationship detection unit108. In a case where the message information output unit 116 isprovided, the display control unit 118 superimposes a message on theimage by further using information output from the message informationoutput unit 116.

<Display Unit>

The display unit 120 presents the AR application image to a user. The ARapplication image is generated by the display control unit 118. Asdescribed above, the display unit 120 is realized by various displaydevices which are formed integrally with or separately from theinformation processing apparatus 100. The display unit 120 may be, forexample, a display screen using an LCD (Liquid Crystal Display), an OLED(Organic Light-Emitting Diode) or a PDP (Plasma Display Panel), or asee-through type or a non-see-through type HMD (Head Mounted Display)which is installed by a user.

As described above, in the information processing apparatus 100according to the present embodiment, a display target object is selectedaccording to a positional relationship between a real object and theinformation processing apparatus 100. Also, the display target object isselected from a virtual object group including a plurality of kinds ofvirtual objects related to the real object. In addition, here, the“plurality of kinds of virtual objects” are virtual objects where asuperimposed image is generated according to a plurality of pieces ofimage data (in other words, original data) which are different from eachother. The plurality of pieces of image data are stored in the object DB112, such as, for example, an “image which looks as if it exists in aninner space of a real object” or an “image which looks as if itprotrudes from a real object”. In addition, the “virtual object” is anobject indicating additional information (annotation) for a real objectas described above.

1-3. Example of Display Change

Next, an example of the display change according to the presentembodiment will be described with reference to FIGS. 3 to 5. FIG. 3 is adiagram illustrating a first state where a virtual object is displayedby the information processing apparatus according to the firstembodiment of the present disclosure. FIG. 4 is a diagram illustrating asecond state where a virtual object is displayed by the informationprocessing apparatus according to the first embodiment of the presentdisclosure. FIG. 5 is a diagram illustrating a change in display of avirtual object by the information processing apparatus according to thefirst embodiment of the present disclosure.

FIG. 3 shows the first state where a virtual object is displayed by theinformation processing apparatus 100 according to the presentembodiment. As described above, in the present embodiment, the kind ofvirtual object which is a display target is selected according to anangle of a real object relative to the information processing apparatus100. In the shown example, in the first state, since an angle A1 of thereal object obj_r (a book) relative to the information processingapparatus 100 is relatively small, a virtual object obj_v1 (an imageregarding a content of the book) which looks as if it exists in an innerspace of the real object obj_r is displayed. In the followingdescription, display of the virtual object when the real object obj_r isreflected in the image in a state close to front view is referred to as“front view display”.

Here, an angle A of the real object obj_r relative to the informationprocessing apparatus 100 may be expressed as a tilt of a posture of thereal object obj_r from a reference posture when a line of sightdirection from the image pickup unit 102 of the information processingapparatus 100 is used as a reference. The reference posture of a realobject is, for example, a posture which is defined in advance for eachreal object, and may be a posture of a real object according to theabove-described model coordinate system. In the real object obj_r (abook) in the shown example, a posture where the cover thereof faces theimage pickup unit 102 is set as the reference posture. Here, since thereal object obj_r has a plate shape, an angle A is defined as an angleformed by a line perpendicular to the surface and a line of sight fromthe image pickup unit 102. In addition, a shape of the real object obj_ris not limited to the plate shape, and may be any shape. Therefore, theangle A of the real object obj_r relative to the information processingapparatus 100 may be defined variously in addition to the shown example.

FIG. 4 shows the second state where a virtual object is displayed by theinformation processing apparatus 100 according to the presentembodiment. In the shown example, in the second state, since an angle A2of the real object obj_r (a book) relative to the information processingapparatus 100 is relatively large, a virtual object obj_v2 (an imageregarding a content of the book) which looks as if it protrudes from thereal object obj_r is displayed. In the following description, thedisplay of virtual object when the real object obj_r is reflected in theimage in a state of oblique view is referred to as “oblique viewdisplay”.

As such, in the present embodiment, the virtual object obj_v1 and thevirtual object obj_v2 of which display positions relative to the realobject obj_r are different from each other are selectively displayedaccording to a positional relationship between the informationprocessing apparatus 100 and the real object obj_r. In the shownexample, the virtual object obj_v1 and the virtual object obj_v2 havedifferent positions in the depth direction when the surface of the realobject obj_r is used as a reference surface. More specifically, in acase where the real object obj_r is reflected in the image in a stateclose to front view, the virtual object obj_v1 which looks as if it isreflected in the real object obj_r and exists therein is displayed. Onthe other hand, in a case where the real object obj_r is reflected inthe image in a state of oblique view, the virtual object obj_v2 whichlooks as if it protrudes from the surface of the real object obj_r isdisplayed.

Thereby, a user can be presented the virtual object obj_v which iseasily visually recognized and where a relationship with the real objectobj_r is also easily understood, so as to be suitable for each postureeven in a case where a posture of the real object obj_r reflected on animage is variously changed. In addition, here, although an example wherea display target object is selected from a virtual object groupincluding the virtual object obj_v1 in the front view display and thevirtual object obj_v2 in the oblique view display has been described,virtual objects included in the virtual object group are not limited tothese two kinds. For example, more kinds of virtual objects may beincluded in the virtual object group, and display target objects may bechanged in more steps according to a posture relationship between a realobject and the information processing apparatus 100. Further, thechanges of the positional relationship between the virtual and realobject may be performed linearly or non-linearly.

FIG. 5 shows an example of the change in display of a virtual object bythe information processing apparatus 100 according to the presentembodiment. As described above, in the present embodiment, display of avirtual object by the information processing apparatus 100 is varieddepending on an angle of a real object relative to the informationprocessing apparatus 100. In addition, in the example described here, anangle A of a plate-shaped real object obj_r relative to the informationprocessing apparatus 100 is defined as an angle formed by a lineperpendicular to the surface of the real object obj_r and a line ofsight from the image pickup unit 102 of the information processingapparatus 100. FIG. 5 is a diagram illustrating a positionalrelationship between the information processing apparatus 100 and thereal object obj_r through simplification only in one direction on thetwo-dimensional plane in relationships between the angle A and the frontview display and the oblique view display of the virtual object. As willbe readily apparent to a person skilled in the art, in practice, similarrelationship is established for any direction on a three-dimensionalspace.

In the shown example, when the information processing apparatus 100faces the real object obj_r, the angle A is 0(deg). In a state ofA=0(deg), display of the virtual object is the front view display as ina case of the angle A=A1 shown in FIG. 3. If the information processingapparatus 100 is gradually tilted with respect to the real object obj_rfrom the state and the angle A exceeds a threshold value T1, the displayof the virtual object is changed to the oblique view display as in acase of the angle A=A2 shown in FIG. 4. On the other hand, if theinformation processing apparatus 100 gradually comes close to the frontsurface of the real object obj_r from the state where the display of thevirtual object is the oblique view display and the angle A becomessmaller than a threshold value T2, then the display of the virtualobject is changed to the front view display.

As such, in the present embodiment, the kind of virtual object whichbecomes a display target object is selected using a threshold value setas an index value (the angle A) regarding a positional relationshipbetween the information processing apparatus 100 and the real objectobj_r. In a case where the image pickup unit 102 captures moving imagesand the kind of virtual object is selected for each of images formingconsecutive frames, a different threshold value may be used depending ona virtual object which has been displayed until then, that is, the kindof virtual object selected in an image of the previous frame, such asthe above-described threshold values T1 and T2. As such, by givinghysteresis to the selection of a virtual object, it is possible toprevent reduction in visibility occurring since display of the virtualobject is frequently changed due to vibration of the angle A around athreshold value caused by camera shake, etc.

1-4. Example of Process Flow

Next, with reference to FIGS. 6 and 7, a description will be made of anexample of the process flow performed by the information processingapparatus according to the present embodiment. FIG. 6 is a flowchartillustrating an example of the process flow of image display process inthe information processing apparatus according to the first embodimentof the present disclosure. FIG. 7 is a flowchart illustrating an exampleof the process flow of display target setting process in the informationprocessing apparatus according to the first embodiment of the presentdisclosure.

FIG. 6 is a flowchart illustrating an image display process according tothe present embodiment. The image display process described here is aseries of processes performed until a virtual object is superimposed onthe image so as to be displayed after the information processingapparatus 100 acquires an image for a real space. In the shown example,the image pickup unit 102 captures moving images, and thus the imagedisplay process is repeatedly performed for the respective imagesforming the consecutive frames.

First, in step S101, the image pickup unit 102 acquires an image. Here,the acquired image is an image which is obtained by capturing a realspace and is reflecting a real object obj_r therein.

Next, in step S103, the image recognition unit 104 performs objectrecognition for the acquired image. Here, the image recognition unit 104recognizes the real object obj_r reflected in the image by the use ofthe model data stored in the model DB 106.

Next, in step S105, the positional relationship detection unit 108detects a positional relationship between the information processingapparatus 100 and the real object obj_r. Here, the positionalrelationship detection unit 108 detects an angle A of the real objectobj_r relative to the information processing apparatus 100.

Next, in step S107, the selection unit 110 determines whether or not thepositional relationship between the information processing apparatus 100and the real object obj_r is varied as compared with an image of theprevious frame. The positional relationship is indicated by the angle A.If the positional relationship is varied, then the selection unit 110,in step S109, selects a display target object on the basis of thepositional relationship after being varied. On the other hand, if thepositional relationship is not varied, since the kind of virtual objectselected as a display target object for the image of the previous frameis continuously used, the selection unit 110 skips step S109.

In addition, a display target selection process in step S109 will bedescribed later with reference to FIG. 7.

Next, in step S111, the display target information output unit 114outputs information related to the display target object selected by theselection unit 110 to the display control unit 118.

Next, in step S113, the display control unit 118 superimposes thevirtual object on the image so as to be displayed on the display unit120 according to the information of the display target object outputtedfrom the display target information output unit 114. Thereby, the imageon which the virtual object appropriately selected by the selection unit110 is superimposed is presented to a user as an AR application image.

FIG. 7 is a flowchart illustrating a display target setting processaccording to the present embodiment. The display target setting processdescribed here is a process performed in step S109 of the image displayprocess as described above. In the shown example, the image pickup unit102 captures moving images, and thus the display target setting processmay be also performed for the respective images forming the consecutiveframes.

First, in step S201, the selection unit 110 determines whether or notthe display of virtual object in the previous frame is the front viewdisplay. As described above, the front view display is a display of thevirtual object obj_v1 which looks as if it exists in an inner space ofthe real object obj_r when the real object obj_r is reflected in theimage in a state close to front view.

Here, if the display of the virtual object in the previous frame is thefront view display, then the selection unit 110, in step S203, sets athreshold value T1 as a threshold value of the angle A used to determinea continuous display target. On the other hand, if the display of thevirtual object in the previous frame is not the front view display, thatis, the display is the oblique view display, the selection unit 110, instep S205, sets a threshold value T2 as the threshold value. Asdescribed with reference to FIG. 5, the threshold values T1 and T2 ofthe angle A in the present embodiment has a relationship of T2<T1.Thereby, it is possible to select a display target object havinghysteresis.

Next, in step S207, the selection unit 110 determines whether or not theangle A detected by the positional relationship detection unit 108 isequal to or less than the threshold value set in step S203 or step S205.

Here, if it is determined that the angle A is equal to or less than thethreshold value, that is, the real object obj_r is reflected in theimage in a state close to front view, in step S209, the selection unit110 sets the virtual object in the front view display as a displaytarget. On the other hand, if it is not determined that the angle A isequal to or less than the threshold value, that is, the real objectobj_r is reflected in the image in a state of oblique view, in stepS211, the selection unit 110 sets the virtual object in the oblique viewdisplay as a display target.

As above, the display target setting process finishes, and informationof the set display target object is output to the display control unit118 by the display target information output unit 114.

1-5. Example of Alternative Selection and Message Display

Next, with reference to FIG. 8, an example of the alternative selectionand message display according to the present embodiment will bedescribed. FIG. 8 is a diagram illustrating an example of thealternative selection and message display according to the firstembodiment of the present disclosure.

FIG. 8 shows a diagram for comparison of a case where alternativeselection and message display are performed and a case where they arenot performed. In the case shown in (a) of FIG. 8, the real object obj_ris reflected in the image in a state of oblique view. Therefore, thevirtual object obj_v2 which looks as if it protrudes from the realobject obj_r is displayed.

On the other hand, in the case shown in (b) of FIG. 8, the real objectobj_r is reflected in the image in a state of oblique view in similarmanner as (a) of FIG. 8. However, in (b) of FIG. 8, since the realobject obj_r is reflected close to the end of the screen, if the virtualobject obj_v2 is to be displayed as in (a) of FIG. 8, a display positionof the virtual object obj_v2 would exceed the image effective region.Therefore, the selection unit 110 selects the virtual object obj_v1which can be displayed in the image effective region as a display targetregardless of the determination result obtained for the angle A of thereal object obj_r relative to the information processing apparatus 100.

Thereby, it is possible to prevent the virtual object from not beingvisually recognized by a user because display of the virtual objectexceeds the image region. However, the display of the virtual objectobj_v2 in a state of (b) of FIG. 8 is not necessarily in a state easilyvisually recognized by a user, for example, since the size thereof issignificantly transformed according to the tilt of the surface of thereal object obj_r. In addition, in this state, it is difficult to letthe user know that the virtual object obj_v2 can be displayed as in (a)of FIG. 8.

Therefore, in the shown example, a message msg based on informationgenerated by the message information output unit 116 is superimposed onthe image so as to be displayed. The message msg may be a message whichprompts the user to pay attention to the real object obj_r so as tochange the image region accordingly and be guided to a state where thevirtual object obj_v2 which is to be originally selected can bedisplayed, such as, for example, “LOOK AT THE OBJECT!”. In addition, themessage information output unit 116 may output, as message information,information for designating a display position of the message msg whichis adjusted so as not to overlap the virtual object obj_v1.

In the above example, for example, an image of an icon or the like maybe displayed instead of the message. In addition, a notification may beprovided to a user through lighting or blinking of an LED(Light-Emitting Diode) lamp included in the information processingapparatus 100 or output of audio from the information processingapparatus 100. In other words, the message information output unit 116in the present embodiment is an example of the notification unit whichsends a notification to a user so as to vary an image region when analternative display target object is selected. This notification may beperformed through changes in display by the message information outputunit 116, the display control unit 118, and the display unit 120 as inthe above-described example. The notification also may be performed by avisual notification unit such as a lamp, or may be performed by an audionotification unit such as a speaker.

As above, the first embodiment of the present disclosure has beendescribed. In the first embodiment of the present disclosure, a displayposition of a real object relative to a virtual object is changeddepending on a positional relationship between the real object reflectedin an image and a device capturing the image. Although an angle is usedas the positional relationship in the above-described example, otherpositional relationships may be used. For example, a distance between areal object and a device may be used as the positional relationship. Inthis case, for example, if a distance between a real object and a deviceis shorter than a predetermined distance and thus the real object isdisplayed so as to be relatively large, a virtual object may bedisplayed in the real object, and, conversely, if the real object isdisplayed so as to be relatively small, the virtual object may protrudeoutwardly from the real object.

According to the first embodiment of the present disclosure, forexample, a virtual object which is displayed at a position which givesthe most comfortable viewing can be superimposed on an image so as to bepresented to a user in each case according to a positional relationshipbetween a real object and a device.

2. Second Embodiment

Next, the second embodiment of the present disclosure will be describedwith reference to FIGS. 9 to 11. In the second embodiment of the presentdisclosure, an amount of information displayed by a virtual object ischanged depending on a positional relationship between a real objectreflected in an image and a device capturing the image.

In addition, in a configuration of the second embodiment, parts otherthan parts described in the following are similar to those of the firstembodiment. The configuration of these parts can be sufficientlydescribed by appropriately replacing the description of the firstembodiment according to the following description, and thus repeateddetailed description thereof will be omitted.

Here, an example of a change in display according to the presentembodiment will be described FIGS. 9 to 11. FIG. 9 is a diagramillustrating a first state where a virtual object is displayed by aninformation processing apparatus according to a second embodiment of thepresent disclosure. FIG. 10 is a diagram illustrating a second statewhere a virtual object is displayed by the information processingapparatus according to the second embodiment of the present disclosure.FIG. 11 is a diagram illustrating a change in display of a virtualobject by the information processing apparatus according to the secondembodiment of the present disclosure.

FIG. 9 shows the first state where a virtual object is displayed by theinformation processing apparatus 100 according to the presentembodiment. In the present embodiment, the kind of virtual object whichis a display target is selected according to a distance of a real objectfrom the information processing apparatus 100. In the first state in theshown example, since the distance d1 from the information processingapparatus 100 to the real object obj_r (a signboard) is relatively long,and the real object obj_r reflected in the image is relatively small, avirtual object obj_v3 (only “SALE” is displayed) which displays asmaller amount of information is displayed. In the followingdescription, display of a virtual object when such a smaller amount ofinformation is displayed is referred to as “outline display”.

FIG. 10 shows the second state where a virtual object is displayed bythe information processing apparatus 100 according to the presentembodiment. In the second state in the shown example, since the distanced2 from the information processing apparatus 100 to the real objectobj_r (a signboard) is relatively short, and the real object obj_rreflected in the image is relatively large, a virtual object obj_v4 (thetime period or place is further displayed) which displays a largeramount of information is displayed. In the following description,display of a virtual object when such a larger amount of information isdisplayed is referred to as “detail display”.

FIG. 11 shows an example of the change in display of a virtual object bythe information processing apparatus 100 according to the presentembodiment. As described above, in the present embodiment, the displayof the virtual object by the information processing apparatus 100 ischanged depending on the distance d from the information processingapparatus 100 to the real object. FIG. 11 is a diagram illustrating apositional relationship between the information processing apparatus 100and the real object obj_r through simplification only in one directionon the two-dimensional plane in relationships between the distance d andthe detail display and outline display of the virtual object. As will bereadily apparent to a person skilled in the art, in practice, similarrelationship is established for any direction on a three-dimensionalspace.

In the shown example, when the information processing apparatus 100 isclosest to the real object obj_r, the distance d is 0. In a state of thedistance d=0, display of the virtual object is the detail display as ina case of the distance d=d1 shown in FIG. 9. If the informationprocessing apparatus 100 becomes distant from the real object obj_r fromthe state and the distance d exceeds a threshold value L1, the displayof the virtual object is changed to the outline display as in a case ofthe distance d=d2 shown in FIG. 10. On the other hand, if theinformation processing apparatus 100 gradually comes close to the realobject obj_r from the state where the display of the virtual object isthe outline display, and then the distance d becomes smaller than athreshold value L2, the display of the virtual object is changed to thedetail display.

As such, in the present embodiment, the kind of virtual object whichbecomes a display target object is selected using a threshold value setas a parameter (the distance d) regarding a positional relationshipbetween the information processing apparatus 100 and the real objectobj_r. In a case where the image pickup unit 102 captures moving imagesand the kind of virtual object is selected for the respective imagesforming consecutive frames, a different threshold value may be useddepending on a virtual object which has been displayed until then, thatis, the kind of virtual object selected in the previous frame, such asthe above-described threshold values L1 and L2. In this manner, bygiving hysteresis to the selection of a virtual object, it is possibleto prevent reduction in visibility occurring since display of thevirtual object is frequently changed due to vibration of the distance daround a threshold value caused by camera shake or the like.

As above, the second embodiment of the present disclosure has beendescribed. In the second embodiment of the present disclosure, an amountof information displayed by a virtual object is changed depending on apositional relationship between the real object reflected in an imageand a device capturing the image. Although a distance is used as thepositional relationship in the above-described example, other positionalrelationships may be used. For example, an angle of a real objectrelative to a device may be used as the positional relationship. In thiscase, for example, if a surface of the real object is viewed nearly fromthe front side and thus the display area is large, a virtual objectdisplaying more information may be displayed, and, conversely, if thesurface of the real object is viewed obliquely and thus the display areais small, a virtual object displaying less information may be displayed.

According to the second embodiment of the present disclosure, forexample, a virtual object which effectively displays an amount ofinformation which gives the most comfortable viewing can be superimposedon an image so as to be presented to a user in each case according to apositional relationship between a real object and a device.

3. Third Embodiment

Subsequently, the third embodiment of the present disclosure will bedescribed with reference to FIGS. 12 to 16. In the third embodiment ofthe present disclosure, the kind of information displayed by a virtualobject is changed depending on a positional relationship between a realobject reflected in an image and a device capturing the image.

In addition, in a configuration of the third embodiment, parts otherthan parts described in the following are similar to those of the firstembodiment. A configuration of these parts can be sufficiently describedby appropriately replacing the description of the first embodimentaccording to the following description, and thus repeated detaileddescription thereof will be omitted.

3-1. Example of Detected Positional Relationship

First, an example of the positional relationship between a real objectand a device detected according to the present embodiment will bedescribed with reference to FIG. 12. FIG. 12 is a diagram illustrating apositional relationship between a real object and a device detectedaccording to the third embodiment of the present disclosure.

FIG. 12 shows four examples (a)-(d) illustrating a detected positionalrelationship in the present embodiment. In the present embodiment, arotation angle B of the real object obj_r reflected in the image isdetected as a positional relationship between the information processingapparatus 100 and the real object obj_r. For example, in a state shownin (a) of FIG. 12, the image pickup unit 102 captures an image includingthe real object obj_r in a state where the information processingapparatus 100 is erected. In the shown example, in this case, therotation angle B is 0(deg). Here, if the information processingapparatus 100 is rotated in a clockwise direction by 90(deg) from thestate shown in (a) of FIG. 12 and enters a state shown in (b) of FIG.12, the real object obj_r is displayed in a state of being rotated bythe rotation angle B=90(deg) from the state shown in (a) of FIG. 12 inthe image which is captured by the image pickup unit 102 and isdisplayed on the display unit 120.

Moreover, when the information processing apparatus 100 is rotated in aclockwise direction and enters a state shown in (c) of FIG. 12, the realobject obj_r is displayed in a state of being rotated by the rotationangle B=180(deg) from the state shown in (a) of FIG. 12 in the imagedisplayed on the display unit 120. On the other hand, when theinformation processing apparatus 100 is rotated in a counterclockwisedirection by 90(deg) from the state shown in (a) of FIG. 12 and enters astate shown in (d) of FIG. 12, the real object obj_r is displayed in astate of being rotated by the rotation angle B=−90(deg) from the stateshown in (a) of FIG. 12 in the image displayed on the display unit 120.

In the present embodiment, the positional relationship detection unit108 of the information processing apparatus 100 detects the rotation ofthe real object obj_r relative to the information processing apparatus100 as the example described above, as a positional relationship of thereal object and the information processing apparatus 100. Morespecifically, the positional relationship detection unit 108 detects arotation of the real object obj_r of which an axis parallel to the imagepickup axis of the image pickup unit 102 is rotated, as a positionalrelationship. Here, the image pickup axis of the image pickup unit 102may be defined, for example, as an optical axis in a portion closest tothe real object obj_r of optical systems of the image pickup unit 102.

In addition, one of differences between the present embodiment and thefirst and second embodiments is that the change in a positionalrelationship between the real object and the information processingapparatus 100 does not substantially change the display of the realobject in an image.

For example, in the examples shown in FIG. 12, it can be said that thedisplay of the real object obj_r is rotated in the coordinate systemusing the information processing apparatus 100 as a reference. However,in many cases, the changes in the states as shown in (a)-(d) of FIG. 12occur in practice by a user changing a method of holding the informationprocessing apparatus 100. In these cases, since the informationprocessing apparatus 100 is rotated itself in a direction reverse to therotation of the display of the real object obj_r in the display unit120, the real object obj_r visually recognized by a user is notsubstantially rotated. For example, in the example shown in (c) of FIG.12, the real object obj_r displayed on the display unit 120 is rotatedby the rotation angle B=180(deg); however, if the information processingapparatus 100 itself is held by a user in a state of being rotated by180(deg) from the initial posture, the display of the real object obj_rvisually recognized by the user is the same as that shown in (a) of FIG.12.

In other words, the present embodiment indicates that the techniquerelated to the present disclosure is effective even in a case where itis not necessarily difficult for a user to visually recognize a realobject or a virtual object due to a change in a positional relationshipbetween the real object and the information processing apparatus 100.

3-2. Example of Display Change

Next, an example of the display change according to the presentembodiment will be described with reference to FIGS. 13 to 15. FIG. 13is a diagram illustrating a first state where a virtual object isdisplayed by the information processing apparatus according to the thirdembodiment of the present disclosure. FIG. 14 is a diagram illustratinga second state where a virtual object is displayed by the informationprocessing apparatus according to the third embodiment of the presentdisclosure. FIG. 15 is a diagram illustrating a change in display of avirtual object by the information processing apparatus according to thethird embodiment of the present disclosure.

FIG. 13 shows the first state where a virtual object is displayed by theinformation processing apparatus 100 according to the presentembodiment. As described above, in the present embodiment, the kind ofvirtual object which is a display target is selected according to arotation of a real object relative to the information processingapparatus 100. In the first state in the shown example, the informationprocessing apparatus 100 is in a state of being placed horizontally tothe real object obj_r (monitor). In this case, the display content(details) of the real object obj_r is displayed as a virtual objectobj_v5 so as to protrude from the real object obj_r. In the followingdescription, such display of the virtual object is also referred to as“content display”.

FIG. 14 shows the second state where a virtual object is displayed bythe information processing apparatus 100 according to the presentembodiment. In the second state in the shown example, the informationprocessing apparatus 100 is in a state of being placed vertically to thereal object obj_r (monitor). In this case, information related to thedisplay content (details) of the real object obj_r is displayed as avirtual object obj_v6 so as to protrude from the real object obj_r. Inthe following description, such display of the virtual object is alsoreferred to as “related information display”.

FIG. 15 shows an example of the change in display of a virtual object bythe information processing apparatus 100 according to the presentembodiment. As described above, in the present embodiment, the displayof a virtual object by the information processing apparatus 100 ischanged depending on a rotation of a real object relative to theinformation processing apparatus 100. In the present embodiment, a rangeis set to a rotation angle, and the kind of information displayed by avirtual object is changed depending on the range.

In the shown example, the information processing apparatus 100 is in astate of being placed vertically to the real object obj_r, and arotation angle B of the real object obj_r relative to the informationprocessing apparatus 100 is detected using a state where the informationprocessing apparatus 100 is erected as a reference. A range R1 or arange R2 is set for the detected rotation angle B, and the virtualobject is displayed as the content display if the rotation angle B iswithin this range. On the other hand, if the rotation angle B is out ofthis range, a virtual object is displayed as the related informationdisplay.

Here, the range R1 and the range R2 are ranges of a rotation angle ofthe real object obj_r where the information processing apparatus 100 isdetermined as being in a state of being placed horizontally to the realobject obj_r. Therefore, both the range R1 and the range R2 may beranges around, for example, +/−90(deg). In addition, the range R1 isused in a case where display of a virtual object is the content display,and the range R2 is used in a case where a virtual object is the relatedinformation display. In other words, the range R2 where the contentdisplay is continued to be performed in a state where the contentdisplay has already been performed is greater than the range R1 of beingchanged to the content display from a state where the relatedinformation display is performed. As such, by giving hysteresis to theselection of a virtual object, it is possible to prevent reduction invisibility occurring since display of the virtual object is frequentlychanged due to vibration of the rotation angle B around a boundary ofthe range caused by camera shake or the like.

3-3. Example of Process Flow

Next, with reference to FIG. 16, a description will be made of anexample of the process flow performed by the information processingapparatus according to the present embodiment. FIG. 16 is a flowchartillustrating an example of the process flow of display target settingprocesses in the information processing apparatus according to the thirdembodiment of the present disclosure.

FIG. 16 is a flowchart illustrating the display target setting processin the present embodiment. The display target setting process describedhere is a process performed in step S109 of the image display processsimilar to that described with reference to FIG. 6. In the shownexample, the image pickup unit 102 captures moving images, and thus thedisplay target setting process may be also performed for the respectiveimages forming the consecutive frames.

First, in step S301, the selection unit 110 determines whether or notdisplay of the virtual object in the previous frame is the contentdisplay. As described above, the content display is display of thevirtual object obj_v5 which displays a content of the real object obj_rwhen the information processing apparatus 100 is placed horizontally tothe real object obj_r.

Here, if the display of the virtual object in the previous frame is thecontent display, the selection unit 110 sets a range R1 as a range ofthe rotation angle B used to determine a continuous display target instep S303. On the other hand, if the display of the virtual object inthe previous frame is not the content display, that is, the display isthe related information display, the selection unit 110 sets a range R2as the range, in step S305. As described with reference to FIG. 15, theranges R1 and R2 of the rotation angle B in the present embodiment has arelationship that R1 is larger than R1. Thereby, it is possible toselect a display target object having hysteresis.

Next, in step S307, the selection unit 110 determines whether or not therotation angle B of the detected obj_r is within the range set in stepS303 or step S305.

Here, if it is determined that the rotation angle B is within the range,that is, the information processing apparatus 100 is placed horizontallyto the real object obj_r, the selection unit 110, in step S309, sets thevirtual object in the content display as a display target. On the otherhand, if it is determined that the rotation angle B is not within therange, that is, the information processing apparatus 100 is placedvertically to the real object obj_r, the selection unit 110, in stepS311, sets the virtual object in the related information display as adisplay target.

As above, the display target setting process finishes, and informationof the display target object set here is output to the display controlunit 118 by the display target information output unit 114.

As above, the third embodiment of the present disclosure has beendescribed. In the third embodiment of the present disclosure, the kindof information displayed by a virtual object is changed depending on apositional relationship between the real object reflected in an imageand a device capturing the image. Although a rotation angle is used asthe positional relationship in the above-described example, otherpositional relationships may be used. For example, an angle of a realobject relative to a device may be used as the positional relationship.In this case, for example, if a surface of the real object is viewednearly from the front side, a content of the real object may bedisplayed, and, conversely, if the surface of the real object is viewedobliquely, information related to the content of the real object may bedisplayed.

According to the third embodiment of the present disclosure, forexample, by changing a positional relationship between a real object anda device, it is possible to change the kind of information displayed bya virtual object which is displayed with regard to the real object. Forexample, if a positional relationship is intentionally changed by auser, the user can use the change in a posture of the device as aninstruction operation for changing display content. In addition, forexample, if a positional relationship is unintentionally changed, theuser can obtain the different kinds of information which has not beenviewed before.

4. Other Embodiments Regarding Apparatus Configuration

Subsequently, other embodiments of the present disclosure regarding anapparatus configuration will be described with reference to FIGS. 17 and18.

<Example Including Sensor Unit>

FIG. 17 is a block diagram illustrating another example of thefunctional configuration of an information processing apparatusaccording to an embodiment of the present disclosure. Referring to FIG.17, an information processing apparatus 200 includes a sensor unit 222as well as an image pickup unit 102, an image recognition unit 104, amodel DB 106, a positional relationship detection unit 108, a selectionunit 110, an object DB 112, a display target information output unit114, a message information output unit 116, a display control unit 118,and a display unit 120.

The sensor unit 222 includes a motion sensor group which measures amotion of the image pickup unit 102. The motion sensor may be, forexample, an acceleration sensor which measures acceleration of the imagepickup unit 102, a gyro sensor which measures a tilt angle, ageomagnetic sensor which measures the azimuth to which the image pickupunit 102 is oriented, or the like. In addition, the sensor unit 222 mayinclude a GPS (Global Positioning System) sensor which receives a GPSsignal and measures latitude, longitude and altitude of the apparatus.

In the shown example, the positional relationship detection unit 108detects a positional relationship between a real object and theinformation processing apparatus 200 (more specifically, the imagepickup unit 102) on the basis of the motion of the image pickup unit 102acquired by the sensor unit 222. For example, the positionalrelationship detection unit 108 detects a posture of the informationprocessing apparatus 200 on the basis of information of acceleration ortilt angle of the image pickup unit 102. If there is information of aposition of a real object, whether or not the real object is on ahorizontal plane such as the ground, or whether or not the real objectis on a vertical plane such as a wall, it is possible to obtain apositional relationship with the real object by detecting a posture ofthe information processing apparatus 200. In addition, the positionalrelationship detection unit 108, as shown in the figure, may detect apositional relationship using information from the sensor unit 222, andmay detect a positional relationship using information from the imagerecognition unit 104 together.

<Example of Server on Network>

FIG. 18 is a block diagram illustrating still another example of thefunctional configuration of an information processing apparatusaccording to an embodiment of the present disclosure. Referring to FIG.18, an information processing apparatus 300 includes an imagerecognition unit 104, a model DB 106, a positional relationshipdetection unit 108, a selection unit 110, an object DB 112, a displaytarget information output unit 114, a message information output unit116, and a communication unit 324.

In the shown example, the information processing apparatus 300 does notinclude the image pickup unit 102, the display control unit 118, and thedisplay unit 120. The information processing apparatus 300 is, forexample, a server installed on a network, and receives data of an imagecaptured by the image pickup unit of an apparatus which is a client asan input image via the communication unit 324. In addition, theinformation processing apparatus 300 transmits information of a displaytarget object output from the display target information output unit 114and further information of a message output from the message informationoutput unit 116, to the client via the communication unit 324. An imageon which a virtual object is superimposed is generated, and the image isdisplayed in the display control unit of the terminal apparatus usingthe transmitted information.

As such, the information processing apparatus according to theembodiments of the present disclosure may not necessarily include theimage pickup unit 102 or the display unit 120, and has variousconfigurations. For example, in the above-described informationprocessing apparatus 300, the display control unit 118 may be provided,and an image on which a virtual object is superimposed may be generatedand transmitted to the client via the communication unit 324. Further,the information processing apparatus may not include the image pickupunit 102 but include the display unit 120, and superimpose a virtualobject on an image captured by an external device so as to be displayed.Furthermore, the information processing apparatus may not include theimage pickup unit 102 but include the display unit 120, and transmit animage where a virtual object is superposed on a captured image to anexternal device.

5. Supplement

<Hardware Configuration>

Finally, with reference to FIG. 19, a description will be made of ahardware configuration of the information processing apparatus 900capable of realizing the information processing apparatus according tothe embodiments of the present disclosure. FIG. 19 is a block diagramillustrating a hardware configuration of the information processingapparatus.

The information processing device 900 includes a CPU (Central ProcessingUnit) 901, ROM (Read Only Memory) 903, and RAM (Random Access Memory)905.

Further, the information processing device 900 may include a host bus907, a bridge 909, an external bus 911, an interface 913, an inputdevice 915, an output device 917, a storage device 919, a drive 921, aconnection port 923, and a communication device 925. The informationprocessing device 900 may include a processing circuit such as DSP(Digital Signal Processor) in addition to or instead of the CPU 901.

The CPU 901 functions as an arithmetic processing unit and a controlunit, and controls the entire operation within the informationprocessing device 900 or a part thereof in accordance with variousprograms recorded on the ROM 903, the RAM 905, the storage 919, or theremovable recording medium 927. The ROM 903 stores programs, operationparameters, and the like used by the CPU 901. The RAM 905 temporarilystores programs used in the execution of the CPU 901, parameters thatchange as appropriate during the execution, and the like. The CPU 901,the ROM 903, and the RAM 905 are mutually connected by a host bus 907constructed from an internal bus such as a CPU bus. Further, the hostbus 907 is connected to the external bus 911 such as a PCI (PeripheralComponent Interconnect/Interface) via the bridge 909.

The input device 915 is a device used by a user such as, for example, amouse, a keyboard, a touch panel, a button, a switch, or a lever. Theinput device 915 may be, for example, a remote control device that usesinfrared rays or other radio waves, or an external connection device 929such as a portable phone corresponding to the operation of theinformation processing device 900. The input device 915 includes aninput control circuit that generates an input signal based oninformation input by a user and outputs the input signal to the CPU 901.The user can, by operating the input device 915, input various data tothe information processing device 900 or instruct the informationprocessing device 900 to perform a processing operation.

The output device 917 includes a device that can visually or audiblyinform a user of the acquired information. The output device 917 can be,for example, a display device such as an LCD (liquid crystal display), aPDP (Plasma Display Panel) an organic EL (Electro-Luminescence) display;an audio output device such as a speaker or headphones; or a printerdevice. The output device 917 outputs the result obtained through theprocessing of the information processing device 900 as text or videosuch as an image or as sound such as voice or audio.

The storage device 919 is a device for storing data, constructed as anexample of a storage unit of the information processing device 900. Thestorage device 919 includes, for example, a magnetic storage device suchas HDD (Hard Disk Drive), a semiconductor storage device, an opticalstorage device, or a magneto-optical storage device. This storage device929 includes, for example, programs or various data executed by the CPU901 or various data acquired from the outside.

The drive 921 is a reader/writer for a removable recording medium 927such as a magnetic disk, an optical disc, a magneto-optical disk, orsemiconductor memory, and is incorporated in or externally attached tothe information processing device 900. The drive 921 reads informationrecorded on a removable recording medium 927 that is mounted, andoutputs the information to the RAM 905. The drive 921 also writesinformation to the removable recording medium 927 that is mounted.

The connection port 923 is a port for directly connecting a device tothe information processing device 900. The connection port 923 can be,for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or aSCSI (Small Computer System Interface) port. In addition, the connectionport 923 may be an RS-232 port, an optical audio terminal, or a HDMI(High-Definition Multimedia Interface) port. When the externalconnection device 929 is connected to the connection port 923, theinformation processing device 900 and the external connection device 929can exchange various data.

The communication device 925 is, for example, a communication interfaceincluding a communication device or the like for connection to acommunications network 931. The communication device 925 can be, forexample, a wired or wireless LAN (Local Area Network) or a communicationcard for Bluetooth (registered trademark) or WUSB (Wireless USB).Alternatively, the communication device 925 may be a router for opticalcommunication, a router for ADSL (Asymmetric Digital Subscriber Line),or a modem for various communication. The communication device 925transmits or receives signals or the like via the Internet or to/fromother communication devices, for example, using a predetermined protocolsuch as TCP/IP. In addition, the communications network 931 connected tothe communication device 925 is a network connected by wire orwirelessly, and is, for example, the Internet, a home LAN, infraredcommunication, radio wave communication, or satellite communication.

The image pickup device 933 is, for example, an apparatus which capturesa real world and generates a captured image by using image sensor suchas CCD (Charge Coupled Device) or CMOS (Complementary Metal OxideSemiconductor) and various components such as lens for picking up asubject image to the image sensor. The image device 933 may beconfigured to pick up still images or moving images.

The sensor 935 may be various types of sensors such as an accelerationsensor, a gyro sensor, a geomagnetic sensor, an optical sensor, and anacoustic sensor. The sensor 935 acquires information related to thestate of an information processing apparatus 900 such as the shape ofhousing of the information processing apparatus 900 and informationrelated to surrounding environment of the information processingapparatus 900 such as brightness or noise in surroundings of theinformation processing apparatus 900. Moreover, the sensor 935 mayinclude a GPS (Global Positioning System) sensor which receives a GPSsignal and measures latitude, longitude and altitude of the apparatus.

An example of the hardware configuration of the information processingapparatus 900 has been described. The respective components describedabove may be configured using general purpose elements, and may beconfigured by hardware specialized to the function of the respectivecomponents. Such configurations can be appropriately changed accordingto the technical level at the time of implementing the presentembodiments.

Although preferred embodiments of the present disclosure are describedin detail above with reference to the appended drawings, the disclosureis not limited thereto. It should be understood by those skilled in theart that various modifications, combinations, sub-combinations andalterations may occur depending on design requirements and other factorsinsofar as they are within the scope of the appended claims or theequivalents thereof.

Additionally, the following configurations are also within the technicalscope of the present disclosure.

(1) An apparatus including:

-   -   an image processor configured to        -   receive a video including an object,        -   determine a positional relationship between the apparatus            and the object,        -   change a positional relationship between an image            superimposed on the video and the object when the positional            relationship between the apparatus and the object changes.

(2) The apparatus according to (1), wherein the image processordetermines an angle between a direction of the object and a direction ofthe apparatus and changes the positional relationship between the imagesuperimposed on the video and the object based on the angle.

(3) The apparatus according to (2), wherein the image processordetermines that the positional relationship between the imagesuperimposed on the video and the object is a front view relationshipwhen the angle is below a first threshold angle.

(4) The apparatus according to (3), wherein the image processor changesthe positional relationship between the image superimposed on the videoand the object from the front view relationship to an oblique viewrelationship when the angle exceeds the first threshold angle.

(5) The apparatus according to (4), wherein the image processor changesthe positional relationship between the image superimposed on the videoand the object from the oblique relationship to the front viewrelationship when the angle drops below a second threshold angle.

(6) The apparatus according to (5), wherein the second threshold angleis lower than the first threshold angle.

(7) The apparatus according to (4), wherein the image processorsuperimposes the image on the video in a plane of the object for thefront view relationship and superimposes the image on the video abovethe object for the oblique view relationship.

(8) The apparatus according to (1) to (7), wherein the image processorsuperimposes a text message on the video with the image.

(9) The apparatus according to (8), wherein the image processorsuperimposes a text message on the video with the image when the imageis modified to fit on a display screen.

(10) The apparatus according to (9), wherein the image processorsuperimposes a text message providing instructions to eliminatemodification of the image to fit on the display screen.

(11) The apparatus according to (1), wherein the image processordetermines a distance between the object and the apparatus and changesthe positional relationship between the image superimposed on the videoand the object based on the distance.

(12) The apparatus according to (11), wherein the image processorchanges the image superimposed on the video based on the distance.

(13) The apparatus according to (12), wherein the image processorchanges the image superimposed on the video based on the distance suchthat additional detail is displayed when the distance is less than afirst threshold distance.

(14) The apparatus according to (13), wherein the image processorchanges the image superimposed on the video based on the distance suchthat additional detail is removed when the distance exceeds a secondthreshold distance.

(15) The apparatus according to (14), wherein the second thresholddistance is shorter than the first threshold distance.

(16) The apparatus according to (1) to (15), further comprising:

-   -   a receiver configured to receive the image from an external        device.

(17) The apparatus according to (1) to (16), further comprising:

-   -   a display configured to display the image superimposed on the        video.

(18) The apparatus according to (1) to (17), further comprising:

-   -   a sensor configured measure positional information of the        apparatus.

(19) A method including:

-   -   receiving a video including an object,    -   determining a positional relationship between the apparatus and        the object,    -   changing a positional relationship between an image superimposed        on the video and the object when the positional relationship        between the apparatus and the object changes.

(20) A non-transitory computer readable medium encoded with a programthat, when loaded on a processor, causes the processor to perform amethod including:

-   -   receiving a video including an object,    -   determining a positional relationship between the apparatus and        the object,    -   changing a positional relationship between an image superimposed        on the video and the object when the positional relationship        between the apparatus and the object changes.

REFERENCE SIGNS LIST

-   100, 200, 300 Information processing apparatus-   102 Image pickup unit-   104 Image recognition unit-   108 Positional relationship detection unit-   110 Selection unit-   114 Display target information output unit-   116 Message information output unit-   118 Display control unit-   120 Display unit-   222 Sensor unit-   324 Communication unit-   obj_r Real object-   obj_v Virtual object

The invention claimed is:
 1. A device comprising: a sensor configured tomeasure motion of the device; a display configured to display a virtualobject; and circuitry configured to acquire an image data of a realspace including an object; detect an angle between a direction of theobject and a direction of the device based on the acquired image data;determine a display mode of the virtual object based on the anglebetween the direction of the object and the direction of the device; andcontrol the display to display the virtual object based on thedetermined display mode; wherein the display mode is selected from aplurality of display modes including a first display mode and a seconddisplay mode; in the first display mode, the virtual object is displayedwith an appearance of being in a plane of the object; in the seconddisplay mode, the virtual object is displayed with an appearance ofprotruding from the object; and the first display mode and the seconddisplay mode are displayed at different times, wherein the acquiredimage data includes data defining a shape of an object.
 2. The device ofclaim 1, wherein the plurality of display modes further includes a thirddisplay mode and a fourth display mode; in the third display mode, afirst amount of information of the object is displayed; and in thefourth display mode a second amount of information on the object isdisplayed, the second amount of information being greater than the firstamount of information.
 3. The device of claim 1, wherein the sensorincludes at least one of a gyro sensor, a geomagnetic sensor, and anacceleration sensor.
 4. The device of claim 1, wherein the positionalrelationship between the device and the object is determined furtherbased on motion information of the device measured by the sensor.
 5. Thedevice of claim 1, wherein the display mode is determined by comparingthe angle to a threshold angle.
 6. A method of using a device having asensor, a display, and circuitry, the method comprising: measuringmotion of the device, using the sensor; displaying a virtual object onthe display; acquiring, using the circuitry, an image data of a realspace including an object; detecting, using the circuitry, an anglebetween a direction of the object and a direction of the device based onthe acquired image data; determining, using the circuitry, a displaymode of the virtual object based on the angle between the direction ofthe object and the direction of the device; and controlling, thedisplay, using the circuitry, to display the virtual object based on thedetermined display mode; wherein the display mode is selected from aplurality of display modes including a first display mode and a seconddisplay mode; in the first display mode, the virtual object is displayedwith an appearance of being in a plane of the object; in the seconddisplay mode, the virtual object is displayed with an appearance ofprotruding from the object; and the first display mode and the seconddisplay mode are displayed at different times, wherein the acquiredimage data includes data defining a shape of an object.
 7. Anon-transitory, computer-readable medium storing instructions that, whenexecuted on a device having a sensor, a display, and circuitry, controlthe device to carry out a method comprising: measuring motion of thedevice; displaying a virtual object; acquiring an image data of a realspace including an object; detecting an angle between a direction of theobject and a direction of the device based on the acquired image data;determining a display mode of the virtual object based on the anglebetween the direction of the object and the direction of the device; anddisplaying the virtual object based on the determined display mode;wherein the display mode is selected from a plurality of display modesincluding a first display mode and a second display mode; in the firstdisplay mode, the virtual object is displayed with an appearance ofbeing in a plane of the object; in the second display mode, the virtualobject is displayed with an appearance of protruding from the object;and the first display mode and the second display mode are displayed atdifferent times, wherein the acquired image data includes data defininga shape of an object.